Image processing device, imaging processing method, and program

ABSTRACT

[Object] To reproduce an image to which an effect desired by a user is added. 
     [Solution] There is provided an image processing device including: an image reverse stabilization processing unit configured to add an effect of expressing shaking to an image on the basis of shaking information on shaking of the image. In addition, there is provided an image processing device including: an image reverse stabilization processing unit configured to decide a degree of an effect of expressing shaking to be added to an image, on the basis of an expectation value of an immersed feeling of an observer with respect to the image.

The present application is a Continuation of application Ser. No.15/318,477, filed Dec. 13, 2016, which is a National Stage Entry ofPCT/JP2015/062301, filed Apr. 22, 2015, and claims priority to JapanesePatent Application JP 2014-136698 filed in the Japanese Patent Office onJul. 2, 2014, the entire contents of which is hereby incorporated byreference.

TECHNICAL FIELD

The present disclosure relates to an image processing device, an imageprocessing method, and a program.

BACKGROUND ART

Camera shaking correction technology for digital cameras and the likehas been already common as described, for example, in PatentLiterature 1. A known example of the camera shaking correctiontechnology is an optical technique of detecting camera shaking with agyro-sensor mounted on an image shooting device, and then driving acorrection lens to move the optical axis in such a direction thatcancels the camera shaking. An electronic camera shaking correctiontechnique is also known which detects the camera shaking in a shotimage, and then cuts out areas for uniform object areas.

CITATION LIST Patent Literature

-   Patent Literature 1: JP 2009-272890A

DISCLOSURE OF INVENTION Technical Problem

The above-described camera shaking correction technology makes itpossible to record a stable image even when an image is shot in anenvironment in which the image shooting device is shaken. Meanwhile, animage subjected to camera shaking correction could possibly fail tosufficiently reproduce a sense of realism felt at the time of imageshooting. The above-described technology can select whether to correctcamera shaking at the time of image shooting, but it is difficult tochange the selection about whether to apply camera shaking correctionafter an image is shot and recorded. It is not possible to sufficientlyreproduce the stableness or a sense of realism as a user desires whenthe image is reproduced.

Accordingly, the present disclosure proposes a novel and improved imageprocessing device, image processing method, and program that canreproduce an image to which an effect desired by a user is added.

Solution to Problem

According to the present disclosure, there is provided an imageprocessing device including: an image reverse stabilization processingunit configured to add an effect of expressing shaking to an image onthe basis of shaking information on shaking of the image.

In addition, according to the present disclosure, there is provided animage processing device including: an image reverse stabilizationprocessing unit configured to decide a degree of an effect of expressingshaking to be added to an image, on the basis of an expectation value ofan immersed feeling of an observer with respect to the image.

In addition, according to the present disclosure, there is provided animage processing method including: adding, by a processor, an effect ofexpressing shaking to an image on the basis of shaking information onshaking of the image.

According to the present disclosure, there is provided an imageprocessing method including: deciding, by a processor, a degree of aneffect of expressing shaking to be added to an image, on the basis of anexpectation value of an immersed feeling of an observer with respect tothe image.

In addition, according to the present disclosure, there is provided aprogram for causing a computer to execute: a function of adding aneffect of expressing shaking to an image on the basis of shakinginformation on shaking of the image.

In addition, according to the present disclosure, there is provided aprogram for causing a computer to execute: a function of deciding adegree of an effect of expressing shaking to be added to an image, onthe basis of an expectation value of an immersed feeling of an observerwith respect to the image.

Advantageous Effects of Invention

According to the present disclosure as described above, it is possibleto reproduce an image to which an effect desired by a user is added.

Note that the effects described above are not necessarily limitative.With or in the place of the above effects, there may be achieved any oneof the effects described in this specification or other effects that maybe grasped from this specification.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic block diagram illustrating a functionalconfiguration example of an image processing system according to a firstembodiment of the present disclosure.

FIG. 2 is a flowchart illustrating an example of recording processing inthe first embodiment of the present disclosure.

FIG. 3 is a flowchart illustrating an example of reproduction processingin the first embodiment of the present disclosure.

FIG. 4 is a block diagram illustrating a functional configuration forexecuting recording processing in the image processing system accordingto the first embodiment of the present disclosure in more detail.

FIG. 5 is a block diagram illustrating a functional configuration of ademultiplexer and decode module included in the image processing systemaccording to the first embodiment of the present disclosure in moredetail.

FIG. 6 is a block diagram illustrating a functional configuration of animmersed degree calculation module included in the image processingsystem according to the first embodiment of the present disclosure inmore detail.

FIG. 7 is a block diagram illustrating a functional configuration of animage reverse stabilization module included in the image processingsystem according to the first embodiment of the present disclosure inmore detail.

FIG. 8 is a block diagram illustrating a functional configuration of adisplay module included in the image processing system according to thefirst embodiment of the present disclosure in more detail.

FIG. 9 is a diagram for further describing processing of calculating animmersed degree in the image processing system according to the firstembodiment of the present disclosure.

FIG. 10 is a diagram illustrating a first example of image display inthe first embodiment of the present disclosure.

FIG. 11 is a diagram illustrating a second example of image display inthe first embodiment of the present disclosure.

FIG. 12 is a schematic block diagram illustrating a functionalconfiguration example of an image processing system according to asecond embodiment of the present disclosure.

FIG. 13 is a flowchart illustrating an example of recording processingin the second embodiment of the present disclosure.

FIG. 14 is a flowchart illustrating an example of reproductionprocessing in the second embodiment of the present disclosure.

FIG. 15 is a schematic block diagram illustrating a functionalconfiguration example of an image processing system according to a thirdembodiment of the present disclosure.

FIG. 16 is a block diagram illustrating a hardware configuration exampleof an image processing device according to an embodiment of the presentdisclosure.

MODE(S) FOR CARRYING OUT THE INVENTION

Hereinafter, (a) preferred embodiment(s) of the present disclosure willbe described in detail with reference to the appended drawings. In thisspecification and the appended drawings, structural elements that havesubstantially the same function and structure are denoted with the samereference numerals, and repeated explanation of these structuralelements is omitted.

The description will be now made in the following order.

1. First Embodiment

1-1. Functional configuration

1-2. Processing flow

1-3. Detailed functional configuration of each unit

1-4. Processing of calculating immersed degree

1-5. Display example

2. Second Embodiment

3. Third Embodiment

4. Hardware configuration

5. Supplemental information

1. First Embodiment

(1-1. Functional Configuration)

FIG. 1 is a schematic block diagram illustrating the functionalconfiguration example of an image processing system according to a firstembodiment of the present disclosure. FIG. 1 illustrates that an imageprocessing system 100 includes a signal capture and processing module110, an encode and multiplexer module 120, a memory module 130, ademultiplexer and decode module 140, an immersed degree calculationmodule 150, an image reverse stabilization module 160, and a displaymodule 190. The immersed degree calculation module 150 receives modesetting 170 and audience environment setting 180. Each component will bediscussed below in detail.

The functional configuration of the above-described image processingsystem 100 may be implemented, for example, by a single image processingdevice or may be dispersively implemented by a plurality of imageprocessing devices. For example, the overall functional configuration ofthe above-described image processing system 100 may be implemented in aterminal device such as a digital camera or a smartphone or tabletequipped with a camera. In this case, an image shot by a terminal deviceand subjected to reverse stabilization processing can be viewed on theterminal device itself. The memory module 130 may be built in a terminaldevice or may be a removable recording medium.

Meanwhile, for example, the functional configuration of the imageprocessing system 100 may be dispersively implemented by a terminaldevice and a server device. In this case, for example, the signalcapture and processing module 110 and the display module 190 may beimplemented in a terminal device, and the functional configuration inbetween, that is, the encode and multiplexer module 120, the memorymodule 130, the demultiplexer and decode module 140, the immersed degreecalculation module 150, and the image reverse stabilization module 160may be implemented by one or more server devices. The communicationbetween the terminal device and a server device, and the communicationbetween the server devices are performed via a variety of wired orwireless networks including the Internet, Wi-Fi, and Bluetooth(registered trademark).

Some of the encode and multiplexer module 120, the memory module 130,the demultiplexer and decode module 140, the immersed degree calculationmodule 150, and the image reverse stabilization module 160 may beimplemented in terminal devices. In this case, a terminal device thatimplements the signal capture and processing module 110 is differentfrom a terminal device that implements the display module 190. Forexample, the signal capture and processing module 110 is implemented ina digital camera, and the display module 190 may be implemented in apersonal computer different from the digital camera.

(1-2. Processing Flow)

FIG. 2 is a flowchart illustrating an example of recording processing inthe first embodiment of the present disclosure. FIG. 2 illustrates thatthe signal capture and processing module 110 first captures the motionof an image shooting device (S101), and captures an audio and an image(S102). Furthermore, the signal capture and processing module 110stabilizes the image in accordance with the motion acquired in S101(S103), and the encode and multiplexer module 120 encodes andmultiplexes the audio and the image (S104). Memory accessing (S105) thencauses the data of the audio and image to be stored in the memory module130.

Here, in the present embodiment, the vector indicating the movementamount of a frame image in the image stabilization in S103 is encodedand multiplexed along with the data of the stabilized image (S104), andstored in the memory module 130 (S105). The vector is used in theprocessing at the time of reproduction discussed below.

FIG. 3 is a flowchart illustrating an example of reproduction processingin the first embodiment of the present disclosure. FIG. 3 illustratesthat the demultiplexer and decode module 140 first demultiplexes anddecodes the data of the image and audio read out from the memory module130 (S121). At this time, the data of the stabilization vector stored inthe memory module 130 along with the data of the image and audio is readout, and decoded and demultiplexed.

Next, the immersed degree suitable for the display of an image to bereproduced is decided before reverse stabilization. The definition ofthe immersed degree will be discussed below. The immersed degreecalculation module 150 determines a mode on the basis of the modesetting 170 (S122). If the mode is “manual,” the immersed degreecalculation module 150 manually sets the immersed degree (S123). Thevalue of the immersed degree set here can be, for example, a value inputthrough a user operation. Meanwhile, if the determination in S122 showsthat the mode is “auto,” the immersed degree calculation module 150analyzes the image (S124) and automatically calculates the immerseddegree (S125).

The image reverse stabilization module 160 reverse stabilizes the imageon the basis of the immersed degree automatically calculated by theimmersed degree calculation module 150 in S125 or the immersed degreemanually set by the immersed degree calculation module 150 in S123(S126). Furthermore, the display module 190 displays the reversestabilized image (S127).

(1-3. Detailed Functional Configuration of Each Unit)

FIG. 4 is a block diagram illustrating the functional configuration forexecuting recording processing in the image processing system accordingto the first embodiment of the present disclosure in more detail. Theconfigurations of the signal capture and processing module 110 and theencode and multiplexer module 120 will be chiefly described below inmore detail with reference to FIG. 4.

The signal capture and processing module 110 includes a lens 111, agyro-sensor 112, an imager sensor 113, a microphone 114, and an imagestabilization processing unit 115. The imager sensor 113 receives thelight through the lens 111, and generates image data. The gyro-sensor112 detects the vibration of the housing including the lens 111.Shifting a correction lens included in the lens 111 in accordance withthe vibration detected by the gyro-sensor 112 achieves optical camerashaking correction (image stabilization) of moving the optical axis insuch a direction that cancels the shaking. Although not illustrated,optical camera shaking correction may be achieved by shifting the imagersensor 113.

The image stabilization processing unit 115 applies electronic camerashaking correction to the image output from the imager sensor 113(stabilizes the image) in accordance with the vibration detected by thegyro-sensor 112. More specifically, the image stabilization processingunit 115 performs processing of cutting out the area of the output imagefrom the input image in a manner that an area smaller than the area ofthe input image provided from the imager sensor 113 is used as the areaof the output image, and an object area included in the output image isfixed. Here, the image stabilization processing unit 115 may decide anarea to be cut out in accordance with a result obtained by thegyro-sensor 112 detecting the vibration, or may decide an area to be cutout on the basis of the analyzation of the image.

The image stabilized by the image stabilization processing unit 115 isinput into an image encode unit 122 of the encode and multiplexer module120. In the present embodiment, the vector that is provided from thegyro-sensor 112 to the image stabilization processing unit 115 andindicates the vibration of the housing, or the vector corresponding tothe image stabilization processing performed by the image stabilizationprocessing unit 115 (e.g., the vector indicating the deviation of theposition of the output image cut out from the input image from thecenter) is input into a stabilization vector encode unit 121 of theencode and multiplexer module 120. Each of the above-described vectorswill also be referred to as stabilization vector. Furthermore, the audiodata acquired by the microphone 114 and corresponding to the image isinput into an audio encode unit 123 of the encode and multiplexer module120.

Here, the above-described camera shaking correction (imagestabilization) processing can be processing of removing at least part ofthe influence resulting from the shaking of the image shooting device atthe time of image shooting. This processing allows the imagestabilization processing unit 115 to output a stabilized image. In thiscase, the image is an image in which at least part of the influenceresulting from the shaking of the image shooting device is removed. Asdiscussed below, the image reverse stabilization module 160 adds aneffect of expressing shaking to the image, thereby reproducing theshaking of the image shooting device in the image. The stabilizationvector is an example of image shooting device shaking informationindicating the shaking of the image shooting device occurring at thetime of image shooting.

The encode and multiplexer module 120 includes the stabilization vectorencode unit 121, the image encode unit 122, the audio encode unit 123,and a multiplexer 124. As described above, the respective encode unitsencode the image data, the audio data, and the stabilization vectorprovided from the signal capture and processing module 110. Themultiplexer 124 multiplexes the data encoded by each encode unit. Themultiplexed data is stored in a data storage 131 included in the memorymodule 130.

FIG. 5 is a block diagram illustrating the functional configuration of ademultiplexer and decode module included in the image processing systemaccording to the first embodiment of the present disclosure in moredetail. FIG. 5 illustrates that the demultiplexer and decode module 140includes a demultiplexer 141, a stabilization vector decode unit 142, animage decode unit 143, and an audio decode unit 144.

The demultiplexer 141 demultiplexes the data stored by the encode andmultiplexer module 120 in the data storage 131 of the memory module 130,and acquires the data of the stabilization vector, the image data, andthe audio data. The respective decode units decode the image data, audiodata, stabilization vector encoded by the encode and multiplexer module120. This offers a decoded stabilization vector 145, a decoded image(stabilized) 146, and a decoded audio 147.

FIG. 6 is a block diagram illustrating the functional configuration ofthe immersed degree calculation module included in the image processingsystem according to the first embodiment of the present disclosure inmore detail. FIG. 6 illustrates that the immersed degree calculationmodule 150 includes a motion analyzation unit 151, an image analyzationunit 152, an audio analyzation unit 153, a display device analyzationunit 154, and an immersed feeling analyzation unit 155.

The motion analyzation unit 151 executes analyzation based on thedecoded stabilization vector 145. The image analyzation unit 152executes analyzation based on the decoded image (stabilized) 146. Theaudio analyzation unit 153 executes analyzation based on the decodedaudio 147. The display device analyzation unit 154 executes analyzationbased on information that is separately acquired and pertains to adisplay device.

The immersed feeling analyzation unit 155 executes analyzation on theimmersed feeling on the basis of a result of the analyzation executed byeach analyzation unit. At this time, the immersed feeling analyzationunit 155 further uses the mode setting (auto/manual) 170, and theaudience environment setting 180 as inputs. The immersed feelinganalyzation unit 155 outputs an immersed degree 157 on the basis of aresult of the analyzation. A specific example of the analyzationprocessing in the immersed feeling analyzation unit 155 will bediscussed below.

FIG. 7 is a block diagram illustrating the functional configuration ofan image reverse stabilization module included in the image processingsystem according to the first embodiment of the present disclosure inmore detail. FIG. 7 illustrates that the image reverse stabilizationmodule 160 includes an image reverse stabilization unit 161. The imagereverse stabilization unit 161 executes image reverse stabilizationprocessing on the basis of the immersed degree 157. The image reversestabilization unit 161 uses the decoded stabilization vector 145 and thedecoded image (stabilized) 146 as inputs. The image reversestabilization unit 161 outputs a reverse stabilized image 162.

Here, the image reverse stabilization module 160 is an example of animage reverse stabilization processing unit that adds an effect ofexpressing shaking to an image on the basis of shaking information onthe shaking of the image. The shaking information is image shootingdevice shaking information indicating the shaking of the image shootingdevice occurring at the time of shooting an image. In other words, theshaking information can include the stabilization vector 145. Asdiscussed below, the image reverse stabilization module 160 decides thedegree of the above-described effect on the basis of the immersed degree157 calculated by the immersed degree calculation module 150. In thepresent embodiment, the immersed degree 157 represents capturingenvironment information on the capturing environment of an image, oraudience environment information on the audience environment of animage.

The image reverse stabilization module 160 can also be an example of animage reverse stabilization processing unit that decides the degree ofthe effect of expressing the shaking to be added to an image on thebasis of the expectation value of the immersed feeling of an observerwith respect to the image. In the present embodiment, the immerseddegree 157 represents the expectation value of the immersed feeling. Asdiscussed below, the immersed degree calculation module 150 decides theimmersed degree 157 on the basis of capturing environment informationindicating the capturing environment of an image, or audienceenvironment information indicating the audience environment of an image.

FIG. 8 is a block diagram illustrating the functional configuration of adisplay module included in the image processing system according to thefirst embodiment of the present disclosure in more detail. FIG. 8illustrates that the display module 190 includes a display I/O 191. Thedisplay I/O 191 displays an image and outputs an audio on the basis ofthe reverse stabilized image 162 and the decoded audio 147.Alternatively, the display I/O 191 transmits an image signal and anaudio signal to an externally connected display device.

(1-4. Processing of Calculating Immersed Degree)

FIG. 9 is a diagram for further describing processing of calculating animmersed degree in the image processing system according to the firstembodiment of the present disclosure. The following describesanalyzation using the motion analyzation unit 151, the image analyzationunit 152, the audio analyzation unit 153, the display device analyzationunit 154, the immersed feeling analyzation unit 155, and the audienceenvironment setting 180 included in the immersed degree calculationmodule 150.

Here, the immersive degree is the expectation value of the immersivefeeling of an observer with respect to a displayed image in the presentspecification. The immersive feeling may be paraphrased, for example, asimpressiveness, a sense of realism, or the like. Images intended to makeobservers feel them more immersing (more impressive or realistic) thushave higher suitable immersed degrees for display. Immersed degreessuitable for displaying images can be calculated, for example, from thedetails of content, scenes, the color of images, and the like.

The motion analyzation unit 151 outputs a value for calculating theimmersed degree in accordance with the direction or magnitude of thestabilization vector (the movement vector of a frame of an image) or thetendency thereof. That is to say, the capturing environment informationrepresented by the immersed degree 157 in the present embodiment caninclude information on the movement vector of a frame of an image. Themotion analyzation unit 151 uses a vector change amount (VC) and aweight for each motion type for analyzation. The vector change amount(VC) is a function for mapping a characteristic such as a change in thevector norm or a change in the angle extracted from the originalstabilization vector to an output value. The weight (MW) for each motiontype is defined in a table generated in advance. A motion type to bedefined is decided, for example, on the basis of a result of motionanalyzation. In the illustrated example, a weight MW1 is set for themotion of “acceleration.” At this time, the motion analyzation unit 151outputs the value of VC*MW1 on the basis of the output VC of the vectorchange amount and the weight MW1 for the motion type.

More specifically, the motion analyzation unit 151 may be set in amanner that, for example, a larger value (VC*MW) is output for greatermotion. In this case, the vector change amount (VC) may be set in amanner that, for example, a larger value is output for greater motion,or a greater weight (MW) may be set for greater motion.

In addition, the motion analyzation unit 151 may be set in a manner thatan output value (VC*MW) changes in accordance with a motion type such asvertical and horizontal movement or rotation. For example, the motionanalyzation unit 151 may be set in a manner that a larger value (VC*MW)is output for vertical and horizontal movement to facilitate reversestabilization, while the motion analyzation unit 151 may be set in amanner that a smaller value (VC*MW) is output for rotating motion tomake reverse stabilization difficult (i.e., an image is stabilized).

The image analyzation unit 152 outputs a value for calculating theimmersed degree in accordance with a characteristic indicated by animage. That is to say, the capturing environment information representedby the immersed degree 157 in the present embodiment can include animage characteristic indicated by an image. The image analyzation unit152 uses an image change amount (IC) and a weight for each scene typefor analyzation. The image change amount (IC) is a function for mappinga characteristic such as a change in the luminance or a change in thecolor distribution extracted from the original input image to an outputvalue. That is to say, the image characteristic can include theluminance or color characteristic of an image in the present embodiment.In addition, the image characteristic may include a scene type. Theweight (SW) for each scene type is defined in a table generated inadvance. A scene type to be defined is decided, for example, on thebasis of a result of scene analyzation. In the illustrated example, aweight SW2 is set for the scene of “ski.” At this time, the imageanalyzation unit 152 outputs the value of IC*SW2 on the basis of theoutput IC of the image change amount and the weight SW2 for the scenetype.

More specifically, the image analyzation unit 152 may be set in a mannerthat, for example, a larger value (IC*SW) is output for a scene such assports or ski to facilitate reverse stabilization. In this case, theimage change amount (IC) may be set in a manner that, for example, achange in the luminance or color distribution of the above-describedscene is considerably reflected, or a greater weight (SW) may be set forthe above-described scene type.

The image analyzation unit 152 may be set in a manner that a largervalue (IC*SW) is output to facilitate reverse stabilization when thecolor distribution considerably changes. Meanwhile, the imageanalyzation unit 152 may be set in a manner that a smaller value (IC*SW)is output to make reverse stabilization difficult when the colordistribution experiences few changes. This is because a change in thecolor distribution is supposed to indicate how drastically the screenchanges. For example, an image shot in a meeting room, where the screenexperiences few changes, has a small change in the color distribution.Meanwhile, an image shot in a place such as a roller coaster, where thescreen drastically changes, has a great change in the colordistribution.

The audio analyzation unit 153 outputs a value for calculating theimmersed degree in accordance with a characteristic indicated by theaudio accompanying an image. That is to say, the capturing environmentinformation represented by the immersed degree 157 in the presentembodiment can include an audio characteristic indicated by the audioaccompanying an image. The audio analyzation unit 153 uses an audiochange amount (AC) and a weight for each audio type for analyzation. Theaudio change amount (AC) is a function for mapping a characteristic suchas a change in the high frequency component energy of the audio or theamplitude of the audio extracted from the original input audio to anoutput value. That is to say, the audio characteristic can include thefrequency component of an audio, the amplitude of an audio, or an audiotype indicated by an audio in the present embodiment. The weight (AW)for each audio type is defined in a table generated in advance. An audiotype to be defined is decided, for example, on the basis of a result ofaudio analyzation. In the illustrated example, a weight AW1 is set forthe scene of “scream.” At this time, the audio analyzation unit 153outputs the value of AC*AW1 on the basis of the output AC of the audiochange amount and the weight AW1 for the audio type.

More specifically, the audio analyzation unit 153 may be set in a mannerthat, for example, a larger value (AC*AW) is output to facilitatereverse stabilization when a noisy audio such as a motor sound and adrift sound is acquired. In this case, the audio change amount (AC) maybe set in a manner that, for example, a change in the frequencycomponent or amplitude of the audio is considerably reflected, or agreater weight (AW) may be set for the above-described audio type.Meanwhile, the audio analyzation unit 153 may be set in a manner that asmaller value (AC*AW) is output to make reverse stabilization difficult(i.e., an image is stabilized) when an audio indicating a quietenvironment is acquired.

The display device analyzation unit 154 uses a device analyzation amount(DA) and a weight for each device type for analyzation. The deviceanalyzation amount (DA) is a function for mapping, for example, the sizeor resolution of the screen of a display device to an output value. Thatis to say, the audience environment information represented by theimmersed degree 157 in the present embodiment can include the size of ascreen on which an image is displayed. The size or resolution of thescreen of a display device can be acquired, for example, frominformation indicating the operation status of the monitor or projectorbuilt in the device or information acquired via an external monitorinterface. The weight (DW) for each device type is defined in a tablegenerated in advance. In the illustrated example, a weight DW3 is setfor the scene of “smartphone.” At this time, the display deviceanalyzation unit 154 outputs the value of DC*DW3 on the basis of theoutput value DA of the device analyzation amount and the weight DW3 forthe device type.

More specifically, the display device analyzation unit 154 may be set ina manner that a smaller value (DA*DW) is output for a screen having alarger size and/or a higher resolution to make reverse stabilizationdifficult. Meanwhile, the display device analyzation unit 154 may be setin a manner that a larger value (DA*DW) is output for a screen having asmaller size and/or a lower resolution to facilitate reversestabilization. This is because when the size of a screen is large or theresolution is high, it is frequently more useful to reduce a burden(such as visually induced motion sickness) on an observer by stabilizingan image rather than to make an image realistic.

A watching analyzation amount (WA) and a weight for each audienceenvironment setting are used for analyzation using the audienceenvironment setting 180. The watching analyzation amount (WA) is afunction for mapping, for example, the distance from an observer to thescreen to an output value. That is to say, the audience environmentinformation represented by the immersed degree 157 in the presentembodiment can include the distance from an observer to a screen onwhich an image is displayed. The weight (EW) for each audienceenvironment setting is defined in a table generated in advance. Thetable can be generated, for example, on the basis of a setting operationof a user. In the illustrated example, a weight EW2 is set for theaudience environment setting of “home.” In the analyzation using theaudience environment setting 180, the value of WA*EW2 is then output onthe basis of the output value WA of the watching analyzation amount andthe weight EW2 for each audience environment setting.

More specifically, the analyzation using the audience environmentsetting 180 may be set in a manner that a smaller value (WA*EW) isoutput to make reverse stabilization difficult as an observer has ashorter distance to the screen. Meanwhile, the analyzation using theaudience environment setting 180 may be set in a manner that a largervalue (WA*EW) is output to facilitate reverse stabilization as anobserver has a longer distance to the screen. This is because when anobserver is close to the screen, it is frequently more useful to reducea burden (such as visually induced motion sickness) on an observer bystabilizing an image rather than to make an image realistic.

The immersed feeling analyzation unit 155 combines results of theanalyzation using the motion analyzation unit 151, the image analyzationunit 152, the audio analyzation unit 153, the display device analyzationunit 154, and the audience environment setting 180. The combination canbe made by adding the output values weighted asVC*MW1+IC*SW2+AC*AW1+DA*DW3+EA*EW2, for example, like the illustratedexample. Furthermore, an immersed degree function (ID) maps the combinedresult to the output value of the immersed degree 157. The immerseddegree 157 is used in the image reverse stabilization module 160 forcontrolling the reverse stabilization of an image.

Here, as illustrated, the immersed degree 157 can be output as acontinuous value. The image reverse stabilization module 160 may decideto what degree the shaking of an image shooting device at the time ofimage shooting is reproduced in an image to be reproduced (the degree ofthe effect of expressing the shaking in an image) by reversestabilization using a stabilization vector, for example, in accordancewith the value of the immersed degree 157. In this case, the imagereverse stabilization module 160 may decide whether to reproduce theshaking of the image shooting device at the time of image shooting in animage to be reproduced (whether to add the effect of expressing theshaking to an image) by reverse stabilization using a stabilizationvector, on the basis of whether the immersed degree 157 exceeds apredetermined threshold. As described above, in the present embodiment,the immersed degree 157 represents capturing environment information onthe capturing environment of an image, or audience environmentinformation on the audience environment of an image. It can also be thussaid that the above-described decision made by the image reversestabilization module 160 is based on the capturing environmentinformation or the audience environment information.

(1-5. Display Example)

FIG. 10 is a diagram illustrating a first example of image display inthe first embodiment of the present disclosure. The example illustratedin FIG. 10 shows display examples of single screens and screens withsub-screens with the mode setting 170 set as auto, manual (OFF), andmanual (ON). In the figure, “R” represents a reverse stabilized image,and “S” represents a stabilized image (not reverse stabilized).

If the mode setting 170 is set as auto, any of a reverse stabilizedimage and a stabilized image is displayed for a single screen, forexample, on the basis of the immersed degree 157 calculated in theabove-described processing of the immersed degree calculation module150. Meanwhile, in the case of a screen with a sub-screen, it isselected to display a reverse stabilized image on the main screen, and astabilized image on the sub-screen in accordance with the immerseddegree 157, or conversely, to display a stabilized image on the mainscreen, and a reverse stabilized image on the sub-screen.

Here, if the immersed degree 157 is greater than a predeterminedthreshold with the mode setting 170 set as auto, a reverse stabilizedimage can be displayed on a single screen, and a reverse stabilizedimage can be displayed on the main screen of a screen with a sub-screen.Meanwhile, if the immersed degree 157 is not greater than thepredetermined threshold, a stabilized image can be displayed on a singlescreen, and a reverse stabilized image can be displayed on thesub-screen of a screen with a sub-screen.

Meanwhile, if the mode setting 170 is set as manual (OFF), a stabilizedimage (not reverse stabilized) is displayed on a single screenirrespective of the immersed degree 157. In this case, the immerseddegree calculation module 150 does not have to execute processing ofcalculating the immersed degree 157.

If the mode setting 170 set as manual (ON), a reverse stabilized imagecan be displayed, irrespective of the immersed degree 157, on a singlescreen, and a reverse stabilized image can be displayed on the mainscreen of a screen with a sub-screen. In this case, the immersed degreecalculation module 150 does not have to execute processing ofcalculating the immersed degree 157. In this example, a manual operationmay switch displaying a reverse stabilized image on the main screen, anda stabilized image on the sub-screen, and conversely displaying astabilized image on the main screen, and a reverse stabilized image onthe sub-screen.

FIG. 11 is a diagram illustrating a second example of image display inthe first embodiment of the present disclosure. In the exampleillustrated in FIG. 11, a reverse stabilized image is displayed on themain screen (Main) of a screen on which a panoramic image is displayedas the sub-screen, and a stabilized image (not reverse stabilized) isdisplayed on the sub-screen (Panorama) on which a panoramic image isdisplayed. Here, panoramic images are frequently used, for example, tolook down on wide areas. Accordingly, the sub-screen may be set todisplay a stabilized image irrespective of the immersed degree 157 likethe illustrated example.

2. Second Embodiment

FIG. 12 is a schematic block diagram illustrating the functionalconfiguration example of an image processing system according to asecond embodiment of the present disclosure. FIG. 12 illustrates that animage processing system 200 includes the signal capture and processingmodule 110, the encode and multiplexer module 120, the memory module130, the demultiplexer and decode module 140, the immersed degreecalculation module 150, the image reverse stabilization module 160, andthe display module 190. The immersed degree calculation module 150receives the mode setting 170 and the audience environment setting 180.

The image processing system 200 according to the present embodimentincludes the similar components to those of the first embodiment, buthas the components differently disposed. In the present embodiment, animage, an audio, and a stabilization vector acquired by the signalcapture and processing module 110 are input to the encode andmultiplexer module 120, and also input to the immersed degreecalculation module 150. The immersed degree calculated by the immerseddegree calculation module 150 is encoded and multiplexed in the encodeand multiplexer module 120 along with the image, the audio, and thestabilization vector, and stored in the memory module 130.

The demultiplexer and decode module 140, which reads out data from thememory module 130, demultiplexes and decodes the data, thereby obtainingthe immersed degree along with the image, the audio, and thestabilization vector. The image, the audio, the stabilization vector,and the immersed degree are input to the image reverse stabilizationmodule 160, and reverse stabilization according to the immersed degreeof the image is executed in the image reverse stabilization module 160.The image and audio processed by the image reverse stabilization module160 are output by the display module 190.

In the present embodiment, the immersed degree has been alreadycalculated before a recorded image is stored in the memory module 130 asdata. Accordingly, it is possible to reverse stabilize the image withoutimage analyzation at the time of reproduction. Such a configuration iseffective, for example, when a device that executes reproductionprocessing does not have a high processing capability, and a device thatexecutes recording processing has a sufficient processing capability.For example, if the mode setting 170 and the audience environmentsetting 180 are differently set at the time of reproduction from at thetime of recording, part of the processing of the immersed degreecalculation module 150 may be executed again at the time of reproductionto update the immersed degree.

FIG. 13 is a flowchart illustrating an example of recording processingin the second embodiment of the present disclosure. FIG. 13 illustratesthat the signal capture and processing module 110 first captures themotion of an image shooting device (S201), and captures an audio and animage (S202). Furthermore, the signal capture and processing module 110stabilizes the image in accordance with the motion acquired in S201(S203).

In the present embodiment, the immersed degree of the image is heredecided. The immersed degree calculation module 150 determines a mode onthe basis of the mode setting 170 (S204). If the mode is “manual,” theimmersed degree calculation module 150 manually sets the immersed degree(S205). The value of the immersed degree set here can be, for example, avalue input through a user operation. Meanwhile, if the determination inS204 shows that the mode is “auto,” the immersed degree calculationmodule 150 analyzes the image (S206) and automatically calculates theimmersed degree (S207).

The encode and multiplexer module 120 encodes and multiplexes the image,the audio, the stabilization vector, and the immersed degree on thebasis of the immersed degree automatically calculated by the immerseddegree calculation module 150 in S207 or the immersed degree manuallyset by the immersed degree calculation module 150 in S205 (S208). Memoryaccessing (S209) then causes the multiplexed data to be stored in thememory module 130.

FIG. 14 is a flowchart illustrating an example of reproductionprocessing in the second embodiment of the present disclosure. FIG. 14illustrates that the demultiplexer and decode module 140 firstdemultiplexes and decodes the data of the image and audio read out fromthe memory module 130 (S221). At this time, the data of thestabilization vector and immersed degree stored in the memory module 130along with the data of the image and audio is read out, and decoded anddemultiplexed.

Next, the image reverse stabilization module 160 reverse stabilizes theimage on the basis of the decoded stabilization vector and the decodedimmersed degree (S222). Furthermore, the display module 190 displays thereverse stabilized image (S223).

3. Third Embodiment

FIG. 15 is a schematic block diagram illustrating the functionalconfiguration example of an image processing system according to a thirdembodiment of the present disclosure. FIG. 15 illustrates that an imageprocessing system 300 includes the signal capture and processing module110, the encode and multiplexer module 120, the memory module 130, thedemultiplexer and decode module 140, the immersed degree calculationmodule 150, the image reverse stabilization module 160, and the displaymodule 190. The immersed degree calculation module 150 receives the modesetting 170 and the audience environment setting 180. In addition, theimage processing system 300 further includes a second memory module 310.

The image processing system 300 according to the present embodiment issimilar to that of the above-described first embodiment in that theimmersed degree calculation module 150 calculates an immersed degree onthe basis of the image, audio, and stabilization vector that are readout from the memory module 130, and demultiplexed and decoded by thedemultiplexer and decode module 140. However, in the present embodiment,the immersed degree calculated by the immersed degree calculation module150 is stored in the second memory module 310. The image reversestabilization module 160 executes reverse stabilization processing onthe basis of the image, audio, and stabilization vector that are decodedby the demultiplexer and decode module 140, and the immersed degree thatis read out from the second memory module 310.

In the present embodiment, the immersed degree calculated by theimmersed degree calculation module 150 is temporarily stored in thememory module 310. Accordingly, it is possible to execute processing ofcalculating the immersed degree before reproducing the image. It is thuspossible to reverse stabilize an image, for example, with no imageanalyzation at the time of reproduction as long as the immersed degreeis calculated through batch processing before a recorded image isreproduced. Such a configuration is effective, for example, when none ofa device that executes recording processing and a device that executesreproduction processing have a high processing capability, andprocessing of calculating the immersed degree is requested from aserver.

Even if a device that executes recording processing and a device thatexecutes reproduction processing have a sufficient processingcapability, requesting processing of calculating the immersed degreefrom a server can save, for example, the battery of a mobile device or awearable device. In that case, the configuration according to thepresent embodiment can be effective.

4. Hardware Configuration

Next, the hardware configuration of an image processing device accordingto an embodiment of the present disclosure will be described withreference to FIG. 16. FIG. 16 is a block diagram illustrating a hardwareconfiguration example of an image processing device according to anembodiment of the present disclosure. An illustrated image processingdevice 900 can implement, for example, a terminal device and/or serverdevice in the above-described embodiment.

The image processing device 900 includes a central processing unit (CPU)901, a read only memory (ROM) 903, and a random access memory (RAM) 905.In addition, the image processing device 900 may include a host bus 907,a bridge 909, an external bus 911, an interface 913, an input device915, an output device 917, a storage device 919, a drive 921, aconnection port 923, and a communication device 925. Further, the imageprocessing device 900 may include an image shooting device 933 and asensor 935 as necessary. The image processing device 900 may include aprocessing circuit referred to as digital signal processor (DSP) orapplication specific integrated circuit (ASIC) instead of or along withthe CPU 901.

The CPU 901 functions as an operation processor and a controller, andcontrols all or some operations in the image processing device 900 inaccordance with a variety of programs recorded on the ROM 903, the RAM905, the storage device 919, or a removable recording medium 927. TheROM 903 stores a program, an operation parameter, and the like which areused by the CPU 901. The RAM 905 primarily stores a program which isused in the execution of the CPU 901 and a parameter which isappropriately modified in the execution. The CPU 901, the ROM 903, andthe RAM 905 are connected to each other by the host bus 907 including aninternal bus such as a CPU bus. In addition, the host bus 907 isconnected to the external bus 911 such as a peripheral componentinterconnect/interface (PCI) bus via the bridge 909.

The input device 915 is a device which is operated by a user, such as amouse, a keyboard, a touch panel, a button, a switch, and a lever. Theinput device 915 may be, for example, a remote control device usinginfrared light or other radio waves, or may be an external connectiondevice 929 such as a mobile phone operable in response to the operationof the image processing device 900. The input device 915 includes aninput control circuit which generates an input signal on the basis ofinformation input by a user and outputs the input signal to the CPU 901.By operating the input device 915, a user inputs various types of datato the image processing device 900 or requires a processing operation.

The output device 917 includes a device capable of visually or audiblynotifying the user of acquired information. The output device 917 mayinclude a display device such as a liquid crystal display (LCD), aplasma display panel (PDP) and an organic electro-luminescence (EL)display, an audio output device such as a speaker and a headphone, and aprinter. The output device 917 may output a result obtained from theprocessing of the image processing device 900 in a form of an image suchas text and an image, and an audio such as an audio and acoustics.

The storage device 919 is a device for data storage which is configuredas an example of a storage unit of the image processing device 900. Thestorage device 919 includes, for example, a magnetic storage device suchas a hard disk drive (HDD), a semiconductor storage device, an opticalstorage device, or a magneto-optical storage device. The storage device919 stores a program to be executed by the CPU 901, various types ofdata, various types of data acquired from the outside, and the like.

The drive 921 is a reader/writer for the removable recording medium 927such as a magnetic disk, an optical disc, a magneto-optical disk, and asemiconductor memory, and is built in the image processing device 900 orexternally attached thereto. The drive 921 reads out informationrecorded in the removable recording medium 927 attached thereto, andoutputs the read-out information to the RAM 905. In addition, the drive921 writes record into the mounted removable recording medium 927.

The connection port 923 is a port used to directly connect a device tothe image processing device 900. The connection port 923 may include,for example, a universal serial bus (USB) port, an IEEE1394 port, and asmall computer system interface (SCSI) port. The connection port 923 mayfurther include an RS-232C port, an optical audio terminal, ahigh-definition multimedia interface (HDMI) (registered trademark) port,and so on. The connection of the external connection device 929 to theconnection port 923 makes it possible to exchange various types of databetween the image processing device 900 and the external connectiondevice 929.

The communication device 925 is, for example, a communication interfaceincluding a communication device or the like for a connection to acommunication network 931. The communication device 925 may be, forexample, a communication card for a wired or wireless local area network(LAN), Bluetooth (registered trademark), a wireless USB (WUSB) or thelike. In addition, the communication device 925 may be a router foroptical communication, a router for an asymmetric digital subscriberline (ADSL), a modem for various kinds of communication, or the like.The communication device 925 transmits a signal to and receives a signalfrom, for example, the Internet or other communication devices on thebasis of a predetermined protocol such as TCP/IP. In addition, thecommunication network 931 connected to the communication device 925 maybe a network connected in a wired or wireless manner, and is, forexample, the Internet, a home LAN, infrared communication, radio wavecommunication, satellite communication, or the like.

The image shooting device 933 is a device that generates a shot image byshooting an image of real space using an image sensor such as a chargecoupled device (CCD) or complementary metal oxide semiconductor (CMOS),as well as various members such as a lens for controlling the formationof an object image on the image sensor, for example. The image shootingdevice 933 may be a device that shoots a still image, and may also be adevice that shoots a moving image.

The sensor 935 includes various sensors such as an acceleration sensor,a gyro sensor, a geomagnetic sensor, an optical sensor, and a soundsensor, for example. The sensor 935 acquires information on the state ofthe image processing device 900, such as the posture of the housing ofthe image processing device 900, and information on an environmentaround the image processing device 900, such as the brightness and noisearound the image processing device 900. The sensor 935 may also includea global positioning system (GPS) sensor that receives GPS signals andmeasures the latitude, longitude, and altitude of the device.

The example of the hardware configuration of the image processing device900 has been described so far. Each of the above-described componentsmay be configured with a general-purpose member, and may also beconfigured with hardware specialized in the function of each component.Such a configuration may also be modified as appropriate in accordancewith the technological level at the time of the implementation.

5. Supplemental Information

The embodiments of the present disclosure may include, for example, animage processing device (a terminal device or a server device) asdescribed above, a system, an information processing method executed bythe image processing device or the system, a program for causing theimage processing device to function, and a non-transitory tangiblemedium having the program recorded thereon.

The preferred embodiment(s) of the present disclosure has/have beendescribed above with reference to the accompanying drawings, whilst thepresent disclosure is not limited to the above examples. A personskilled in the art may find various alterations and modifications withinthe scope of the appended claims, and it should be understood that theywill naturally come under the technical scope of the present disclosure.

Further, the effects described in this specification are merelyillustrative or exemplified effects, and are not limitative. That is,with or in the place of the above effects, the technology according tothe present disclosure may achieve other effects that are clear to thoseskilled in the art based on the description of this specification.

Additionally, the present technology may also be configured as below.

(1)

An image processing device including:

an image reverse stabilization processing unit configured to add aneffect of expressing shaking to an image on the basis of shakinginformation on shaking of the image.

(2)

The image processing device according to (1), wherein

the shaking information includes image shooting device shakinginformation indicating shaking of an image shooting device occurring ata time of shooting the image.

(3)

The image processing device according to (1) or (2), wherein

the image reverse stabilization processing unit decides a degree of theeffect on the basis of capturing environment information on a capturingenvironment of the image.

(4)

The image processing device according to (3), wherein

the image reverse stabilization processing unit decides a degree towhich the effect is added, on the basis of the capturing environmentinformation.

(5)

The image processing device according to (4), wherein

the image reverse stabilization processing unit decides whether to addthe effect, as the degree to which the effect is added.

(6)

The image processing device according to any one of (3) to (5), wherein

the capturing environment information includes information on a movementvector of a frame of the image.

(7)

The image processing device according to any one of (3) to (6), wherein

the capturing environment information includes an image characteristicindicated by the image.

(8)

The image processing device according to (7), wherein

the image characteristic includes a luminance or color characteristic ofthe image.

(9)

The image processing device according to (7) or (8) wherein

the image characteristic includes a scene type.

(10)

The image processing device according to any one of (3) to (9), wherein

the capturing environment information includes an audio characteristicindicated by an audio accompanying the image.

(11)

The image processing device according to (10), wherein

the audio characteristic includes a frequency component of the audio, anamplitude of the audio, or an audio type indicated by the audio.

(12)

The image processing device according to any one of (1) to (11), wherein

the image reverse stabilization processing unit decides a degree of theeffect on the basis of audience environment information on an audienceenvironment of the image.

(13)

The image processing device according to (12), wherein

the audience environment information includes a size of a screen onwhich the image is displayed.

(14)

The image processing device according to (12) or (13), wherein

the audience environment information includes a distance from anobserver to a screen on which the image is displayed.

(15)

The image processing device according to any one of (1) to (14), wherein

the image is an image in which at least part of influence resulting fromshaking of an image shooting device is removed, and the image reversestabilization processing unit adds the effect to the image to reproducethe shaking of the image shooting device in the image.

(16)

An image processing device including:

an image reverse stabilization processing unit configured to decide adegree of an effect of expressing shaking to be added to an image, onthe basis of an expectation value of an immersed feeling of an observerwith respect to the image.

(17)

The image processing device according to (16), wherein

the expectation value of the immersed feeling is decided on the basis ofcapturing environment information indicating a capturing environment ofthe image.

(18)

The image processing device according to (17), wherein

the capturing environment information includes an image characteristicindicated by the image.

(19)

The image processing device according to (17) or (18), wherein

the capturing environment information includes an audio characteristicindicated by an audio accompanying the image.

(20)

The image processing device according to any one of (16) to (19),wherein

the expectation value of the immersed feeling is decided on the basis ofaudience environment information indicating an audience environment ofthe image.

(21)

The image processing device according to (20), wherein

the audience environment information includes a size of a screen onwhich the image is displayed.

(22)

The image processing device according to (21), wherein

the audience environment information includes a distance from anobserver to a screen on which the image is displayed.

(23)

The image processing device according to any one of (16) to (22),wherein

the image reverse stabilization processing unit adds the effect on thebasis of shaking information on shaking of the image.

(24)

An image processing method including:

adding, by a processor, an effect of expressing shaking to an image onthe basis of shaking information on shaking of the image.

(25)

An image processing method including:

deciding, by a processor, a degree of an effect of expressing shaking tobe added to an image, on the basis of an expectation value of animmersed feeling of an observer with respect to the image.

(26)

A program for causing a computer to execute:

a function of adding an effect of expressing shaking to an image on thebasis of shaking information on shaking of the image.

(27)

A program for causing a computer to execute:

a function of deciding a degree of an effect of expressing shaking to beadded to an image, on the basis of an expectation value of an immersedfeeling of an observer with respect to the image.

(28) An image processing device including:

an image reverse stabilization processing unit configured to add aneffect of expressing shaking of an image on the basis of capturingenvironment information on a capturing environment of the image.

(29) An image processing device including:

an image reverse stabilization processing unit configured to add aneffect of expressing shaking of an image on the basis of audienceenvironment information on an audience environment of the image.

REFERENCE SIGNS LIST

-   100, 200, 300 image processing system-   110 signal capture and processing module-   120 encode and multiplexer module-   130 memory module-   140 demultiplexer and decode module-   150 immersed degree calculation module-   160 image reverse stabilization module-   190 display module-   310 second memory module

The invention claimed is:
 1. An image shooting apparatus comprising: animage sensor configured to capture a moving image; and circuitryconfigured to associate a stabilization information to the moving imagein a case where a stabilization process has been applied to the movingimage to produce a stabilized image, and to generate areverse-stabilized image based on the stabilization information and thestabilized image, wherein the reverse-stabilized image has a largerdegree of movement than the stabilized image.
 2. The image shootingapparatus according to claim 1, wherein the stabilization process is anoptical camera shaking correction.
 3. The image shooting apparatusaccording to claim 1, wherein the stabilization process is applied tothe moving image by signal processing circuitry.
 4. The image shootingapparatus according to claim 1, wherein the stabilization informationrepresents a movement vector.
 5. The image shooting apparatus accordingto claim 1, wherein the circuitry is configured to associate an immerseddegree to the moving image.
 6. The image shooting apparatus according toclaim 1, wherein the reverse-stabilized image includes a shaking effect.7. The image shooting apparatus according to claim 1, further comprisinga memory configured to store at least one of the stabilizationinformation or the moving image.
 8. The image shooting apparatusaccording to claim 1, wherein the image sensor is a CCD device or a CMOSdevice.
 9. An image shooting method comprising: capturing, by an imagesensor, a moving image; associating, by circuitry, a stabilizationinformation to the moving image in a case where a stabilization processhas been applied to the moving image to produce a stabilized image; andgenerating, by the circuitry, a reverse-stabilized image based on thestabilization information and the stabilized image, wherein thereverse-stabilized image has a larger degree of movement than thestabilized image.
 10. The image shooting method according to claim 9,wherein the stabilization process is an optical camera shakingcorrection.
 11. The image shooting method according to claim 9, whereinthe stabilization process is applied to the moving image by signalprocessing circuitry.
 12. The image shooting method according to claim9, wherein the stabilization information represents a movement vector.13. The image shooting method according to claim 9, further comprising:associating, by the circuitry, an immersed degree to the moving image.14. The image shooting method according to claim 9, wherein thereverse-stabilized image includes a shaking effect.
 15. The imageshooting method according to claim 9, further comprising: storing, by amemory, at least one of the stabilization information or the movingimage.
 16. A non-transitory computer-readable medium storinginstructions that, when executed by a processor associated with an imageshooting device, cause the image shooting device to perform operationscomprising: capturing, by an image sensor of the image shooting device,a moving image; associating, by circuitry of the image shooting device,a stabilization information to the moving image in a case where astabilization process has been applied to the moving image to produce astabilized image; and generating, by the circuitry, a reverse-stabilizedimage based on the stabilization information and the moving image,wherein the reverse-stabilized image has a larger degree of movementthan the stabilized image.